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data and university-based research - the latter ranking 15th of 20 influencing factors. 
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Consistent with Beck’s (1994; 1997) risk theory, we theorize that education policy making 
in both contexts is influenced by both macro- and micro-level factors, where choice of 
policy evidence is mediated by personal considerations and political risk factors. This 
suggests a weak policy development paradigm that is resistant to independent research- 
informed evidence. 

Keywords: education; school boards; ministries of education; government; research; 
evidence; decision-making; policy; risk theory. 

Investigacion y evidencia cientifica en la toma de decisiones educativas: una 
comparacion de los resultados de dos estudios pan-canadienses 
Resumen: En este articulo comparamos el uso que se le ha dado a la investigacion y otras 
evidencias cientificas en la toma de decisiones de dos grupos de elite de politica educativa, 
provenientes de dos contextos distintos -los ministerios de educacion provinciales y los 
distritos escolares. Los datos se derivan de dos estudios pan-canadienses: Galway (2006) y 
Sheppard, Galway, Brown and Wiens (2013). Los resultados muestran que las decisiones 
pollticas a nivel ministerial estan principalmente influenciadas por factores politicos y 
pragmaticos, creencias personales y profesionales y conocimiento local. El papel de la 
investigacion externa se demuestra que es relativamente marginal y se reduce a estudios 
cuantitativos y evaluaciones de desempeno. Los responsables de la toma de decisiones a 
nivel de distrito escolar estan menos influenciados por los factores politicos y pragmaticos 
y conflan mas en las creencias, valores y experiencias personales junto con el 
asesoramiento de profesionales y los resultados de investigaciones internas. Los resultados 
de ambos estudios demuestran una escasa confianza en los datos externos y la 
investigacion universitaria -que ocupa el puesto numero 15 de 20 en el ranking de factores 
influyentes. En consonancia con la teoria del riesgo de Beck (1994; 1997), sostenemos que 
la formulacion de pollticas educativas en ambos contextos se ve influenciada por factores 
macro y micro, donde la eleccion de la evidencia politica esta mediatizada por 
consideraciones personales y factores de riesgo politico. Esto supone un debil desarrollo 
del modelo politico que es resistente a las pruebas procedentes de investigacion 
independiente. 

Palabras-clave: educacion; consejos escolares; ministerios de educacion; gobierno; 
investigacion; evidencia; toma de decisiones; politica; teoria del riesgo. 

Pesquisa e dados cientlficos na toma de decisoes educacionais: uma compara^ao entre os 
resultados de dois estudos pan-canadenses. 

Resumo: Neste trabalho, comparamos o uso que tem sido dado a investigate e outras evidencias 
cientificas na tomada de decisoes de dois gmpos da elite politica em educa^ao, de dois contextos 
diferentes -o ministerios provinciais de educa^ao e distritos escolares. Os dados foram obtidos de 
dois estudos pan-canadenses: Galway (2006) e Sheppard, Galway, Brown e Wiens (2013). Os 
resultados mostram que as decisoes pollticas a nivel ministerial sao influenciados principalmente por 
fatores politicos e pragmaticos, cren^as pessoais e profissionais e conhecimento local. O papel da 
investigate* externa mostra que e relativamente marginal, reduzida para estudos quantitativos e 
avalia^oes de desempenho. Os responsaveis pela tomada de decisoes em nivel de distrito escolar sao 
menos influenciados por fatores politicos e pragmaticos e confiam mais nas cren^as, valores e 
experiencias pessoais com aconselhamento profissional e os resultados das investigates internas. 

Os resultados de ambos estudos demonstram uma falta de confianza nos dados externos e pesquisa 
academica, que e classificada como a No. 15 de 20 no ranking dos fatores influentes. Em 
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consonancia com a teoria do risco Beck (1994; 1997), que argumentam que a formula^ao de politicas 
educacionais em ambos os contextos e influenciada por fatores macro e micro, onde a escolha de 
evidencia polltica e influenciada por considera^oes pessoais e fatores de risco polltica. Este e um 
fraco desenvolvimento do modelo politico que e resistente a evidencia de pesquisa independente. 
Palavras-chave: eelucacao; conselhos escolares; Ministerios da Educacao; governo; pesquisa; 
provas; tomada de decisao; polltica; teoria do risco. 

Introduction 

We begin this paper with a short account of three incidents, which brought us to the 
realization that education policy making is a very public enterprise, where beliefs, opinions and 
politics are significant policy considerations. Several years ago, at the beginning of the school year, 
the front-page story in a major provincial newspaper bemoaned the fact that Francophone students 
had performed very poorly in provincial examinations and there was some speculation that the 
school board was placing linguistic concerns ahead of subject area competence in the recruitment of 
teachers. As the issue unfolded, and despite the absence of any supporting research, this 
‘conventional wisdom’ seemed to become accepted as professional knowledge within the K-12 
education community where we both held senior administrative positions. Some time later, one of 
us was purchasing symphony tickets when the artistic director of the orchestra came rushing out of 
his office with program notes for the season’s inaugural concert. In the conversation that ensued, he 
highlighted the British Prime Minister’s plan to redouble Britain’s investment in music education. He 
then volunteered that music education was dying in Canadian schools, a message that later appeared 
in the symphony’s program booklet and was reported in the local media. Then, a few weeks later, at 
a meeting of provincial education ministers, a colleague commented on a story from a regional 
newspaper in France. The article described the French Education Minister’s vision for Inis country’s 
education system, which emphasized the continuing relevance of core subjects and traditional 
methods, such as reading, writing and dictation. He noted that the shopkeeper who sold him the 
paper engaged in a lengthy monologue on what he felt was wrong with the Minister’s plan. 

While, in and of themselves, neither of these is particularly unusual, we were stmek by the 
fact that all three represent very public expressions of interest in the education policy process. These 
incidents highlight the public standing of education in Western society; everyone seems to care 
about education and everyone has an opinion - from parents to shopkeepers, and yes, to symphony 
conductors. But, how do policymakers negotiate these often divergent and competing 
representations? In Canada’s global-competitive environment the ‘performance’ of the education 
system seems to have become a standard against which special interests and the public measure the 
effectiveness of government in addressing public concerns relating to regional and national 
economic downturns, competitiveness and community/regional sustainability. These arrangements 
draw attention to education policy making as a high-stakes undertaking. However, the question of 
what policy makers consider to be valid evidence for policy development has not been fully 
examined and questions continue to be raised about the extent to which government policy is 
grounded in research-based knowledge. 

In this article we begin with a literature review on research evidence and policy/decision¬ 
making in education, focusing on British, Australian and Canadian contexts. In the second section 
we describe the methodologies from two pan-Canadian studies that examine the use of research and 
other evidence in the policy formation practices of two groups of education policy elites, situated in 
different contexts. The first study examined the use of research and evidence by ministry level 
decision makers: education ministers and senior ministry bureaucrats (Galway, 2006). The second 
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study considered the factors and evidence used by decision makers (school board trustees and 
superintendents) at the school district level (Sheppard, Galway, Brown & Wiens, 2013). In the last 
section we discuss and compare the findings and, following on earlier work in this area, offer 
suggestions for (1) improving the integration of research into the policy practices of education 
decision makers, and (2) strengthening the intersection of researchers, education practitioners, and 
policy makers. 


Research Evidence and Education Policy 

Since the 1970s there has been considerable focus on the impact of research on the 
production of social, health and educational policy (e.g., Caplan 1979; Cooper, Levin & Campbell, 
2009; Cooper, 2013, 2014; Edwards, Sebba & Rickinson, 2007; Honig & Coburn, 2008; James & 
Pollard, 2011; Landry, Amara & Lamari, 2001; Lavis, Lomas, Hamid & Sewankambo, 2006; Levin & 
Riffle, 1997; Lindblom, 1980; Lindblom & Woodhouse, 1993; Majone, 1989; Martin, Downe, Grace 
& Nutley, 2010; Moat, Lavis, Wilson, Rottingen & Barnighausen, 2013; Nutley, Jung & Walter, 2008; 
Nutley, Walter & Davis, 2003, 2009; Pollard, 2007, 2008; Qi & Levin, 2013; Rosenblatt & Tseng, 
2010; Stone, 2002; Strike, 2006; Tseng & Seidman, 2007; Weiss, 1979; Yohalem & Tseng, 2015). 
While many social science researchers perceive their raison d’etre to be connected to the 
development of sound public policy decisions, policy-making in education is seen to be somewhat 
resistant to the influence of research-based knowledge and, driven by other, more immediate political 
and pragmatic factors (Davis, 1999; Honig & Coburn, 2008; Ince, 2008; Levin, 2004a; Reimer & 
McGinn, 1997). The problem of the so-called research-policy divide has been approached from 
several perspectives and while it has become widely accepted that educational research has improved 
in volume and quality, difficulties with effective dissemination and integration into practice and 
underutilization by policy elites remain vexed problems (Cooper, 2013, 2014; Honig & Coburn, 2008; 
Qi & Levin, 2013; Yohalem & Tseng, 2015). 

Policy theorists have long argued that decisions about education should be tied more closely 
to research evidence. There is a growing push for ministers and other public sector policy makers to 
refer more of their policy decisions to independent research and these calls have been growing louder 
in recent years (DeBroucker & Sweetman, 2001; Fitz-Gibbon, 2003; Honig & Coburn, 2008; 
Naughton, 2005; Slavin, 2002, 2004; Thomas & Pring, 2004; Zussman, 2003). The challenge has been 
to effectively convey this idea to politicians, the media and the public, who often seem to base 
decisions on their own, different forms of evidence — popular sentiment or their own experiences 
and perceptions - rather than on research evidence (Levin, 2004b). Some researchers have accused 
policymakers of taking their lead from local pressures and political and practical considerations such 
as public opinion, conventional professional wisdom and media reports (e.g., Black, 2001; Cooper & 
Levin, 2010; Kwok, 2003; Leicester, 1999; Levin, 2003a; Ungerleider, 2003; Weissberg, 2002). Others 
have mentioned idiosyncratic factors such as dependencies, loyalties, associations, self-interest, 
personal values, beliefs, experiences, biases and fears (e.g., Honig & Coburn; 2007; Reimer & 
McGinn, 1997; Stone, 2002). Reimer and McGinn (1997, p. xiv) describe some of the problems they 
experienced in their policy work with senior education decision-makers in the international context: 
We started as researchers, eager to get developing countries to improve their 
education systems. We were convinced that our skills in research and analyzing data 
would make an important difference. To our great dismay and chagrin, we found it 
difficult to get decision makers to listen to us. When they did listen, they often were 
critical of what we had to say and seldom followed our advice, it seemed. [...] 

Almost all their comments were negative, denying the validity and usefulness of the 
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reports. ... [T]hey drew up recommendations for action, many of which directly 
contradicted what we had recommended. 

While many in the academy will identify with the frustrations expressed in this excerpt, it may be a 
little naive for researchers to expect policymakers to abandon their own perceptions and agendas in 
favour of a strictly linear technocratic approach to policy development. The seminal work of policy 
researchers such as Weiss (1979) and Lindblom (1980) raised questions about the direct or 
‘instrumental’ influence of research-based evidence on policy, claiming instead that research 
utilization is indirect, long term and circuitous. As Levin (2001) comments, “[t]he political world is 
.. .shaped by beliefs more than facts” (p. 14). A number of scholars have observed that in cases 
where research does impact policy, the effect normally takes place after an extended period - a 
phenomenon described by Weiss (1979) as the ‘enlightenment’ function of research. 

Much of the critique of so-called “evidence-free” policy making can be theorized through the 
notion of a chasm between the academic and political/policy contexts. Nathan Caplan’s (1979) early 
work on “two communities” theory suggested that researchers and policymakers operate in entirely 
different contexts with “different and often conflicting values, different reward systems and different 
languages” (p.459) Caplan (1979, p. 468) argued that the coupling of existing knowledge to user 
needs requires collaborative arrangements whereby the researchers and policy makers come to an 
understanding on “which aspects of a [policy] problem are to be decided on the basis of data-based 
knowledge and which ones are to be decided on the basis of non-research knowledge.” Because 
university-based researchers operate according to their own, different accountability and reward 
systems, this has typically required that they follow their own research agendas while maintaining an 
appropriate level of independence from the government policy environment — even publicly 
criticizing government policy in some circumstances. As Levin (2004b) notes, “university-based 
research is driven by the university culture and reward system; it is primarily aimed at communication 
with other scholars and the rewards are related to research grants and peer recognition” (p. 9). 
Moreover, the engagement of academics in research on specific policy problems, as in the case of 
commissioned research brings its own set of problems including serious ethical dilemmas relating to 
authenticity and independence. Strike (2006) observes that the relationship between government and 
researchers is “fraught with moral peril” (p. 67). On one hand, there is the hope that researchers can 
advance social progress when they serve the needs of policy makers. On the other hand, 
commissioned studies are not subject to the same rigorous review standards as independent research: 
The research done at the behest of policymakers is often not refereed. It is often 
“quick and dirty” - done under time pressures and poorly funded. Moreover, 
policymakers often ask researchers to evaluate or assist in the implementation of 
policies and programs in which the policymakers have a vested interest. They may be 
uninterested in funding research that might fail to support favoured programs. The 
questions may be stmctured so as to assume the merits of current policy and policy 
aspirations. Most troubling is that, at the state level, those policymakers who fund 
research often have influence over the budget and reputation of the public 
universities in their states. Researchers may fear offending public officials, lest their 
access to funds and influence dry up. University administrators may be reluctant to 
have their faculties assume a critical or oppositional role, and policymakers are likely 
to look for sympathetic researchers. (Strike, 2006, p. 67) 

Government actors, for their part, have charged researchers with ignoring important policy areas in 
education (Blunkett, 2000; Galway, 2006). In the past, some critics have suggested that, with the 
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possible exception of commissioned research, much of the independently-produced university 
research may be inaccessible or does not provide the answers decision makers are looking for when 
deciding among policy options (Cooper & Levin, 2010; Lindblom & Woodhouse, 1993; Neilson, 
2001; Pring, 2000; Stone, 2002). In the mid-1990s an intense debate began in the UK around the 
quality and utilization of educational research. The mckus was propelled by an article in the Times 
Education Supplement , wherein Alan Smithers (1995) argued that the only criterion for judging the value 
of educational research should be the extent of its usefulness to policy makers and practitioners 
(Dunseath, 2000). Hargreaves (1996, p. 7) later declared that education research was ‘second-rate’ and 
decried its inability to “make a serious contribution to fundamental theory or knowledge,” charging 
instead that it cluttered up obscure academic journals, was irrelevant to practice and was 
uncoordinated with antecedent research. Two controversial British reviews of the quality and 
utilization of educational research followed. Tooley and Darby (1998) evaluated the nature and 
quality of educational research in Britain; based on a sampling of articles published in four British 
educational journals, the researchers concluded that only about one third of the articles could be said 
to have been based on sound research practice. Citing weaknesses in methodology, policy relevance, 
objectivity and quality of argument, the authors suggested that education research must become non¬ 
partisan, pragmatic and relevant in its design, include a plan for dissemination, and be guided by an 
overall research strategy. 

The Tooley and Darby study was followed closely by a second review by Hillage, Pearson, 
Anderson and Tamkin (1998). Hillage and his colleagues put forward the widely contested view that 
the research-policy divide was due, in large part to an education research agenda that was ‘supply- 
driven’; that is, the research originates with researchers and not policymakers and practitioners, a 
model described by Landry and his colleagues as the ‘science push model’ (Landry et al., 2001). They 
argued further that education research tended to be backward looking and evaluative, rather than 
forward-looking and exploratory, resulting in negligible policy impact. This idea overlaps with the 
notion that education research lacked a ‘cumulative character;’ whereby new studies help to constmct 
a knowledge base by building effectively on earlier work (Davies 1999; Hammersley, 1998; Ince, 
2008). 

While some of the observations in the Tooley and Darby and Hillage reports have been 
without serious challenge several have been contested. Hodgkinson (1998) criticized Tooley and 
Darby (1998) for making a fundamental distinction between empirical research and non-empirical 
research and for evaluating qualitative studies by predominantly quantitative, positivist-realist criteria. 
He further charged that the review ignored basic ontological and epistemological precepts of research 
conducted in post-industrial traditions; for example, the fact that many qualitative researchers do not 
claim that their work is generalizable in a statistical sense, nor do they claim objectivity. 

Several reviews describing the state of education research in Australia have been less critical 
of the research community. Bates (2001) undertook a meta-analysis of research reviews in Great 
Britain and Australia over the preceding ten-year period. He concluded that the Australian reviews, 
in comparison to those conducted in Great Britain, tended to deliver much more positive 
assessments of the value of education research, both in terms of its international reputation and its 
use among teachers and policymakers. Ribbins, Bates and Gunter (2003) account for this difference, 
in part, by the nature of the reviews. While the Australian work has tended to begin with educational 
practice to inform research, the UK reviews have tended to begin with the research and work 
forward to practice. 

Between 1999 and 2000 five reviews focusing on research and practice in the Australian 
context were undertaken as part of an overall initiative sponsored by the Australian Department of 
Education, Training and Youth Affairs (DETYA) and the Australian Research Council. The first 
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concluded that much of the Australian research base was devoted to issues of teaching and learning 
(Holbrook, Ainley, Bourke, Owen, McKenzie, Mission & Johnson, 2000). Connections between 
university researchers and the educational community were considered to be robust and included 
ministry-faculty research committees, the publication of research in teacher-focused publications and 
the secondment of academics to ministry staff positions. There was additional evidence, based on 
claims by practitioners, that findings from education research had permeated practice, although the 
review acknowledged that research utilization was often mediated through individual policy 
advocates. In a second study, Figgis, Zubrick, Butorac and Anderson (2000) examined the impact of 
research on policy, starting at the policy/program level and working backwards to determine whether 
selected policies/programs were founded on a research base. Their work put forward the notion that 
practitioners and policymakers were connected to research through ‘connecting webs’ or networks 
with multiple nodes. They suggested that the application of research in an educational context 
requires human intervention of motivated actors and the active marketing of research knowledge by 
its producers. A third study examined teacher practice to determine the extent to which it was 
reflective of a research base and concluded that research was, in fact, a substantial influence among 
several ‘proximal’ sources of influence on teaching practice (McMeniman, Cumming, Wilson, 
Stevenson & Sim, 2000). The fourth study examined citation rates of Australian researchers in 
international journals (Phelan, Anderson & Bourke, 2000). This bibliometric analysis demonstrated 
that Australian research was extensively cited in a range of knowledge domains including pedagogy, 
administration and policy development. A fifth study examined the influence of research on policy 
decision-making within the vocational education and training system in Australia (Selby-Smith, 1999). 
Of the senior education decision makers surveyed, two thirds claimed that strategic and political 
considerations were most influential in decision-making while research was often used to justify or 
support a decision already taken, variously described as the symbolic use of research (Landry et al., 
2001), use of research to mobilize support (Nutley et al., 2003; Weiss, 1998) and use of research to 
build political support, (Honig & Coburn, 2008). Selby Smith (1999) made several conclusions with 
respect to the authenticity of research for policy decision-making in education. First, research does 
impact decision-making, but it exerts its influence on the broader long-term education agenda - 
extending to the so-called ‘big ideas’ in education. He suggested that researchers be modest in their 
expectations about the impact of research on decision-making noting that “research is only one of a 
number of sources of information available to decision-makers and information from all sources is 
only one of many inputs into the decision-making process” (p. 556). Second, because decision makers 
bring their own experiences and backgrounds to bear on their practice, “the extent of research impact 
is affected by the knowledge, attitudes and experience of decision-makers” (p. 565). Finally, to be 
influential, research must be conducted in an environment of collaboration where robust linkages 
exist between decision makers and researchers at all stages of the research process. An “emphasis on 
linkages rather than dissemination alone increases the mutual responsibility of the parties, since 
enduring linkages are based on sustained mutual esteem, and understanding of the potential 
contribution of each party, and a commitment to collaboration for the good of the system” (p. 565). 

Prior to the mid-2000s there were relatively few references to the quality of educational 
research within the Canadian context. Most of the discussion is framed around the scope of the 
research effort, which was generally found to be somewhat lacking. Levin (2003a), for example, 
acknowledged pockets of strength, but claimed that the educational research enterprise generally, is 
small, short-term, narrow and fragmented. Although the research effort within faculties of education 
was growing, he concluded that there were few organized mechanisms to enable education research 
to be translated into knowledge for policy. Applied research in Canada, according to Levin {ibid.) had 
remained within the purview of government departments and third parties, such as privately funded 
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institutes and think tanks. Willinsky (2003) similarly observed that the influence of academic research 
in Canada “pales before the impact of the government’s own sources of information, whether from 
politicians, bureaucrats, or government professionals” (p. 2). He noted, however, that the availability 
of academic articles through electronic means has facilitated research access among policy advisors 
and decision makers and this has acted as a “counterforce to policymakers’ reliance on a small 
number of academic consultants as gatekeepers and sources for research” (p. 1). According to 
Landry, et al., (2001) part of the problem was that the body of work on research utilization was 
disparate - scattered across different social science disciplines and recorded in disconnected journals. 
There were also problems with narrow and overly prescriptive definitions of research utilization 
which had come to be associated with direct instmmental or linear uses of research rather than 
conceptualizing research use as a broader, more interactive and circuitous process. 

In 2003, deBroucker and Sweetman compiled a series of papers by well-known Canadian 
educators and researchers with the goal of examining and evaluating the impact of Canadian 
education research impact to date. The authors drew three conclusions about the state of education 
research as policy evidence. First, compared to the health-care sector in Canada, education research is 
used to a far lesser extent; second, there is a pressing need to involve a broader set of people than 
researchers in considering research issues, and finally, there is a need to expand the indicator set to 
include a broader range of measures, including those indicators that are more difficult to measure. 
Shortly thereafter the then federal (Liberal) government established the Canadian Council on 
Learning (CCL) as an independent national agency with a mandate to produce, promote and 
disseminate educational research (Mason, 2012). Before funding was cut by a successor government 
in 2009, the CCL had made good progress in drawing attention to several key policy issues in 
education by completing systematic reviews and disseminating the research findings through a 
number of channels, most notably, an on-line repository of thematic reviews, research and data. 
Cooper et al. (2009) acknowledge a strengthening of capacity and cite several provincial initiatives 
aimed at increasing support for evidence-based policy and practice, but point out that integrated pan- 
Canadian strategies are more difficult because education in Canada is not a federal responsibility. 

From the perspective of raising awareness of research in education, the fallout from these 
reviews and from the discussions and debates that followed was both positive and progressive. More 
recent work in this field has left behind the divisive debates around research quality and attempted to 
focus on defining and stmcturing the field of evidence-informed practice as a research area (Cooper 
et al., 2009; Nutley et al., 2003) defining and promoting high quality and highly relevant educational 
research (Earl, 2011; James & Pollard, 2008; Pollard, 2007, 2008) and understanding the different 
forms and trajectories of research use both in education and other public sectors (Nutley et al., 2008; 
Rosenblatt & Tseng, 2010). A major advance in promoting this agenda in the UK was the Teaching 
and Learning Research Program (TLRC), a large-scale effort to spearhead a series of projects and 
thematic analyses intended to engage users in research-informed practice both to improve research 
quality and generate new knowledge (James & Pollard, 2011; Pollard, 2011). Pollard (2007) remarked 
that the program was intended to support high quality and highly relevant research that could “satisfy 
the ambition of‘Pasteur’s Quadrant’” (p. 640), a reference to Stokes (1997) categorization describing 
a form of research that marries pure basic research to applied research. The ten-year £40 million 
program, which concluded in 2011, generated a series of ten principles of effective pedagogy in four 
broad areas and is widely regarded to have been successful in translating academic research into user- 
friendly forms and engaging users in culturally and politically current educational discourse (Earl, 
2011; Galvin, 2008; Hogan, 2011; James & Pollard, 2011; Pollard, 2008). 

Other researchers have been working on understanding the different forms and pathways of 
research use. While those working in this field have argued about different perspectives on research 
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use (for example, rationalist-decisionist/rational linear versus more ambiguous and interactive 
approaches) there is some agreement that there is a diversity of avenues through which research 
permeates and influences educational practice and pedagogy (Nutley et al., 2008). Landry et al. (2001) 
examined the utilization of social science research in Canada across four disciplines and 
conceptualized the process as cumulative rather than instrumental; dependant on the engagement of 
researchers with end users rather than the quality of the research products. Drawing on research in 
the social care field and cross-sector reviews, Nutley and her colleagues describe several models of 
research use, noting that both rational-linear and interactive perspectives could be found operating in 
each. This suggests that there is value in conceptualizing such models and perspectives as iterative 
and overlapping; their use being appropriate at different times and in different contexts (Nutley et al, 
2008; Nutley et al., 2009). 

A number of recent articles have begun to concentrate on research utilization as ‘evidence 
into practice’ with emphasis on the intersection of researchers and practitioners (Earl, 2011; Edwards 
et al., 2007; Nutley et al., 2003) including linkage mechanisms, research mediation efforts and new 
dissemination strategies (Cooper, 2013; Landry et al., 2001; Martin et al., 2010; Qi & Levin, 2013). We 
have already mentioned the work of Landry and his colleagues. Based on their study of more than 
1200 social science researchers in Canada, they concluded that the most important determinants of 
utilization are “the mechanisms linking the researchers to the users, the users’ context and the 
publication assets of the researchers” (p. 334). Edwards et al. (2007) similarly found that establishing 
reciprocal relationships with users as co-researchers could enhance practitioner value for research. 

As the field has grown and expanded from health to education and to other social sectors 
new terminology has come into usage: knowledge transfer, evidence-informed policy, data-driven 
decision-making and knowledge brokering, to name a few. Knowledge mobilization (KM) has been 
touted as a useful all-encompassing term because it conveys the notion of direction instead of 
random interaction and it “embodies the idea that the use of knowledge is a social process, not just 
an intellectual task” (Cooper et al., 2009, p.166). The more recent North American work on 
education research utilization/research impact has framed the connection of research and policy 
environments in terms of KM and knowledge brokering with governments, schools and school 
districts, and community organizations (Cooper, 2013, 2014; Cooper & Levin, 2010; Cooper et al., 
2009; Jbilou, Amara & Landry, 2007; Knowles et al., 2013; Qi & Levin, 2013, Wilson, Lavis, Traverse 
& Rourke, 2010; Yohalem & Tseng, 2015). 

Honig and Coburn (2008) were interested in which forms of evidence, if any, were used in 
school district offices and the purposes for which the evidence was used. They conducted a 
comprehensive review of relevant research literature but found a large proportion of the material in 
circulation were advocacy pieces leaving only about fifty peer-reviewed publications. The major 
findings suggest that the process of evidence use is complex, highly political and based largely on 
local knowledge. Along the same vein. Cooper and Levin (2010) surveyed a sample of 
superintendents, principals and other leaders to study how research use was perceived, supported and 
integrated into the operations and activities of school districts and schools across Canada. They 
found a strong interest in research use, but a weak and loosely coupled institutional research capacity 
that depended on “a few interested people rather than being deeply embedded in daily practice” (p. 
61 )- \ ^ ‘ 

Since the early 2000s, there has been a proliferation of interest in KM including more 
publications, new networks of researchers and practitioners, and more outlets and spaces for 
interchange on evidence and practice, but also a number of lingering problems (Cooper et al., 2009; 
Qi & Levin, 2013). Notwithstanding the contribution of Nutley et al. (2003) in suggesting a 
framework for organizing the evidence-into-practice research agenda. Cooper and her colleagues 
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complain that that conceptually and theoretically, the field is still in disarray, there are too few 
empirical studies on most of the issues, and efforts to increase research impact have not been 
properly assessed (Cooper et al., 2009). Some of the Canadian work by this group and others at 
Queen’s University and the University of Toronto is attempting to address some of these problems. 
In a recent study of research mediation in Canada, Cooper (2013) takes on the unenviable task of 
disentangling some of the cross-sectorial confusion around KM terminology and knowledge 
brokering models, hypothesizing about how research brokering organizations (RBOs) are situated in 
the education sector and creating a typology of Canadian RBOs (e.g., the Atlantic Institute for Market 
Studies and the Fraser Institute) in education. RBOs are grouped into four categories - government, 
not-for-profit, for-profit and member RBOs - and are defined by their orientation along seven 
common dimensions. This particular work is an important step towards redressing some of the 
conceptual problems in defining the field of KM raised by Cooper and her colleagues in their earlier 
work (Cooper et al., 2009). In an effort to assess KM efforts among knowledge producing 
organizations, Qi and Levin (2009) studied websites from 100 institutions and agencies (including 
governments, universities and faculties of education, and other organizations) in Canada and other 
English-speaking countries. The researchers judged the research sharing effort to be low to modest, 
product-oriented and passive (unidirectional) with no opportunity for two-way interaction. Cooper 
(2014) similarly looked at on-line strategies and social media as a linkage mechanism between 
producers and users of research. The websites of forty-four Canadian RBOs were examined to assess 
their on-line and social media dissemination strategies. The social media effort was considered to be 
modest - mostly small-scale Facebook pages and Twitter networks - and there were few avenues for 
two-way interchange. 

The foregoing discussion highlights the thorny questions of what constitutes evidence for 
education policy and how education research becomes introduced into the policy process. 

Researchers and social policy critics still have divergent and incomplete conceptualizations about the 
relationship between research and practice. While this is an intriguing area, it has not been fully 
interrogated, the literature is still somewhat disconnected, and there are gaps in our knowledge base. 
As Cooper et al. (2009) have observed, after all of the activity related to KM “very little is known 
about how practice organizations, whether governments or schools and school systems, find, share 
and use research.” The analysis presented in this paper attempts to address some of these blank 
spots by addressing three interconnected research questions: (1) what are the factors and influences 
that drive policy decision-making in Canadian education ministries as compared with school boards, 
(2) how important is each category of evidence in relation to the others, and (3) to what extent is 
education research valued as ‘authentic’ policy evidence? 

Research Design 


The 2006 Study 

The 2006 study was a two-phase mixed-methods design with two participant groups: 
Canadian provincial deputy ministers and equivalents (survey/questionnaires) and current or former 
provincial ministers of education (in-depth, semi-structured interviews). 

Survey method. A population of 95 senior education public servants was identified using 
the Canadian Education Association Handbook and contact lists provided by the Council of 
Ministers of Education, Canada (CMEC) and provincial departments of education. A questionnaire 
developed for this research was field tested and administered to deputy ministers and assistant 
deputy ministers of education and their equivalents working in Canadian ministries of education 
using direct and electronic mail. A total of 51 valid questionnaires were returned for a response rate 
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of 53 percent (Quebec was not represented). Demographic information such as age, gender, 
educational background and years of experience in public administration was also collected. In Phase 
II, in-depth, semi-structured interviews were conducted with eight (8) participants - five (5) current 
and three (3) former provincial ministers of education. Participants were selected and contacted 
based on language (English-speaking), probable access, and proximity to the researcher. 

Participants were asked to recall three instances of education policy decisions or policy advice they 
had provided to ministers during the previous 12 months (N=148). They were then asked to rate, on 
a seven point Likert scale, the extent to which each of a series of factors influenced the 
corresponding decision or recommendation. They were then asked to recall examples of research or 
indicators they had recently used in their work. 

Data analysis of numeric and categorical data involved the generation of means and standard 
deviations for Likert scale items and frequency distributions for categorical data, in accordance with 
generally established principles and consistent with those procedures outlined by Dillman (1999), 
Fowler (1998), and Henerson, Morris and Fitz-Gibbon (1987). 

Interview method. Interview questions focused on participant representations of 
research and evidence in education decision-making, in particular (1) how they conceptualize 
education research; (2) the extent to which research and other forms of evidence are utilized in 
decision-making; and, (3) perceptions of accessibility, relevance, utilization and value of 
education research. The interview protocol followed the procedures described by Bok van 
Kammen and Stouthamer-Loeber (1998), Mangione (1998), Marshall and Rossman (1989), and 
Maxwell (1998). All, except one telephone interview, were face-to-face, private interviews and all 
were recorded and transcribed. Interviews ranged in length from approximately 60 minutes to 
more than two hours with an average interview length of approximately 90 minutes. 

The analysis of interview data was modeled after the principles and strategies for interview 
data described by Creswell (1998) and Quinn Patton (1987). First, there was a general review of the 
printed transcripts of each of the interviews and the combined interview data was grouped according 
to question. The transcripts were then analyzed for identifying themes, findings and relationships 
among the data, and notes are recorded in the margins of these texts. The second step of the 
analysis process involved examining the transcripts, and over a period of time, thinking about the 
data sets, and identifying coherent and important examples, themes and patterns in the data, a 
procedure described by Quinn Patton (1987) as content analysis. These ideas and relationships were 
further developed in the form of longer notes and memos that were sorted into categories and 
themes, forming the preliminary findings. As the organization of findings and themes matured, the 
third step of the process involved selecting samples of text from the interviews that were illustrative 
and confirmatory of the finding or theme. In cases where the same idea was expressed repeatedly, 
participant representations were summarized and paraphrased or used as exemplars of the consensus 
of the group. The final data reduction step involved creating displays of information using diagrams 
showing interconnections among ideas and findings while identifying larger global themes running 
through the representations of the participant groups. Finally the interview transcriptions were re¬ 
read carefully to validate or discard these preliminary findings. Following from Silverman’s (2001) 
discussion on avoiding anecdotalism, the data were searched for negative exemplars in the data that 
could be argued to falsify or refute the findings and the categories, themes and findings were revised 
accordingly. 
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The 2012 Study 

For the 2012 study we collected data through the use of a survey/questionnaire and a series 
of focus group interviews. There were three participant groups representing Anglophone school 
boards in all Canadian provinces and territories: (1) school board trustees, (2) school district 
superintendents, and (3) executive directors of each provincial school boards association who were 
also members of the Canadian School Boards Association executive board. 

Participants. The selection of participants for both the survey/questionnaire and 
interviews was conducted with sensitivity to gender, experience, ethnicity and regional 
geography. A total of 369 individuals from across Canada (331 school board trustees and 38 
district superintendents) completed questionnaires. As well, we conducted 21 focus group 
sessions with school board trustees, school district superintendents, and members of the 
Canadian School Boards Association executive board. 

Survey method. The survey/questionnaire was modelled after the one used in the 2006 
study, with some variations. Prior to the distribution of the questionnaire, it was pilot tested 
with a convenience sample of six individuals who had extensive experience as either a 
superintendent of education or trustee. The research protocol was also presented at a session 
with interested members of the Canadian Association for the Study of Educational 
Administration (CASEA) at its 2010 Congress. Minor adjustments to the research instruments 
were made on the basis of the resulting feedback from these validation exercises. 

Each participant was asked to identify specific policy-relevant decisions or recommendations that 
s/he made during the past year. They were then asked to rate, on a seven point Likert scale, the 
extent to which each of a series of factors influenced the corresponding decision or 
recommendation. Our analysis of the survey data is limited to descriptive measures such as 
frequency counts and the calculation of mean scores and standard deviations. For the survey 
analysis, because of low return rates among school district superintendents, it was not possible to 
report separately for school board trustees and superintendents. Data from school board trustees 
and district superintendents are pooled. 

Focus group method. The focus group protocol, and analysis of the focus group data, 
followed the procedures described by Meijer, Verloop, and Beijaard (2002), Merriam (2009), and 
Miles and Fluberman (1994). Focus group sessions were conducted separately for tmstees and 
superintendents. To increase the validity of findings, we took considerable care to collect data from 
school board members and school district superintendents from all provinces and territories and 
from those holding office with the CSBA Board. We also triangulated data collection such that each 
of research team members conducted a number of individual 60-90 minute focus group sessions (5 
to 10 participants per session) that occurred at different times and in differing locations over the 
eighteen-month data collection period. Each focus group session was recorded and transcribed. To 
ensure anonymity of each participant we assigned pseudonyms as required. We analyzed the 
transcribed focus group data through the application of a qualitative data analysis program (QDA 
Miner, 2011). Using this program, we coded separate data segments for school board trustees, 
executive directors of provincial school board associations and school district superintendents of 
education. Subsequently, we retrieved these coded segments and saved them as separate reports that 
facilitated the development of specific themes. Also, we were able to develop frequency counts and 
the percentage of respondents who provided particular responses. It is important to note, however, 
that all focus group participants did not respond to each question. As a consequence, our reporting 
that a specific percentage of the focus group participants held a particular view, for instance, should 
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not be interpreted that a remaining percentage held a contrary view; if no percentage is reported for 
the contrary view, it means the others were silent on the issue. 

Limitations. Although we employed a Likert scale instrument in addition to focus 
group interviews to collect data, this work is primarily dependent on naturalistic and interpretive 
methodologies (Cohen, Manion & Morrison, 2011) that acknowledge personal involvement of 
the researchers and anticipates subjectivity. Data from interviews are actively constituted by the 
intervention of the researcher with research participants. They would otherwise not exist. From 
a theoretical perspective the responses of research participants in this study are seen more as 
representations of perspectives and ‘moral forms’ than simple reports on reality (Silverman, 
2001). Given the pan-Canadian scope of this study and holding realistic expectations of how 
much time we could reasonably expect of study participants at any one point in time, we utilized 
a questionnaire/survey as a means of collecting individual participant perspectives relating to his 
or her role as either a trustee or superintendent. Our analysis of the questionnaire/survey data is 
entirely limited to the determination of frequency counts and mean scores. Accordingly, we 
consider this work to be solely naturalistic and interpretive. 

Ethical review. The research protocol for both studies was approved in advance by a 
university interdisciplinary committee on ethics in human research and followed the principles 
outlined in the Tri-Council Policy Statement on Ethical Conduct for Research involving 
Humans (TCPS2). 


Research Findings 


The 2006 Study 

Profile of participants: Senior bureaucrats. The data show that senior executive positions 
in Canadian education ministries are reserved for highly educated, mature individuals with extensive 
experience in education or public administration, but not necessarily within schools, school boards 
or other educational institutions outside of government. Deputy and assistant deputy minister 
positions tend to be male-dominated - females accounted for less than 40% of education executives 
- and are occupied by mid to late career bureaucrats, who have completed more than one advanced 
degree. As a group, senior bureaucrats have a broad range of administrative experience both within 
and outside of government, yet are generally new to their current positions as education executives. 
More than half the senior officials surveyed were in their current jobs for three or fewer years (Table 
!)• 


Table 1 


Senior Bureaucrats’ Years of Experience in Present Position and in any Administrative Position (N—51) 


Years Experience 

Present Position 

Percent 

Any Admin. Position 

Percent 

0-3 years 

26 

51.0 

3 

5.9 

4-6 years 

12 

23.5 

8 

15.7 

7-9 years 

4 

7.8 

4 

7.8 

10-12 years 

5 

9.8 

9 

17.6 

More than 12 years 

3 

5.9 

26 

51.0 

Not reported 

1 

2.0 

1 

2.0 

Total 

51 

100 

51 

100 
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Categories of policy issues. We first sought to understand the general categories of 
education policy issues addressed by policy actors at the government (ministry of education) 
level. Senior education executives were asked to describe three policy-relevant decisions or 
recommendations they recall having made during the previous 12 months 1 and to rate the 
significance of each policy decision or recommendation on a seven point Likert scale. Data 
describing ‘perceived significance’ are means and standard deviations where +3 represented 
‘very significant,’ 0 was neutral, and -3 represented ‘not at all significant. Participants reported a 
total of 148 policy decisions or recommendations that could be assigned to 16 broad policy 
categories 2 . Results showing the frequency cited and perceived significance, by policy category, 
are presented in Table 2. 

Senior education bureaucrats reported their primary focus was on corporate matters - 
educational finance and resourcing policy, accountability policy and strategic planning. These three 
areas accounted for almost half of the policy decisions/recommendations recalled. The areas least 
cited were: school bussing, professional development, special education, social development and 
equity and education research, together comprising less than seven percent of all policy decision 
and/or recommendations reported. Program and policy issues, such as literacy and post-secondary 
education, aboriginal education, as well as finance and resource issues were judged to be the most 
significant. Also rated as highly significant were a series of related issues — programming, curriculum, 
instruction and assessment, as well as educational planning, governance and accountability. 


Table 2 


Categories, Frequency and Perceived Significance of Policy Decisions Recalled by Senior Bureaucrats (N—148) 


Policy Category 

Frequency 

Cited 

Percent 

Perceived 

Significance 

Standard 

Deviation 

Finance/Education Resources 

41 

28.4 

2.24 

0.81 

Accountability, Testing and Assessment 

15 

10.1 

1.80 

1.15 

Post-secondary Programming 

15 

10.1 

2.07 

0.80 

Educational (Strategic) Planning 

14 

9.5 

2.00 

0.96 

K-12 Curriculum/Instruction 

11 

7.4 

2.00 

1.10 

Career/Labour Market Development 

10 

6.8 

1.80 

1.14 

Literacy / Adult Education Policy 

9 

6.1 

2.44 

1.01 

Education Governance 

8 

5.4 

1.87 

0.83 

Francophone Education/Governance 

5 

3.4 

1.80 

0.45 

Aboriginal Policy 

5 

3.4 

2.20 

0.83 

Education Infrastructure 

4 

2.7 

1.50 

0.90 

Education Research 

3 

2.0 

1.67 

1.53 

Social Development and Equity 

3 

2.0 

-1.00 

0.00 

Special Education 

2 

1.4 

2.00 

0.00 

Professional Development 

1 

0.7 

2.00 

0.00 

School Bussing 

1 

0.7 

2.00 

0.00 

Total 

148 

100.0 

1.99 

0.94 


1 Or since they had been in their current position, if it had been less than 12 months. 

2 In several instances some of the data were combined to form broader categories of factors/influences. 
These are footnoted where they appear in the presentation of results. 
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Evidence and policy practice: Survey data. Data presented in Table 3 were generated 
from participant representations of their recent policymaking practice. Again, senior public servants 
were asked to describe three policy-relevant decisions or recommendations that they recall having 
made during the previous 12 months, ’ and in the context of a given list of factors/evidence, they 
were asked to indicate the relative influence of each of the factors on each decision or 
recommendation. Data were pooled and, for each factor, expressed as the mean plus or minus the 
standard deviation where -3 represented “did not at all influence”, 0 was neutral and 3 represented 
“influenced to a great extent”. 

Findings reveal that greatest importance was placed on internal advice provided by staff and 
colleagues, followed by past experience, personal and professional beliefs, corporate/pragmatic 
evidence 3 4 , and political-democratic considerations 5 . Within the broad category of political 
considerations the data show that the representations of special interest groups was a moderately 
important consideration while claims about the influence of public opinion and the representations 
of the mass media were less definitive. Among the factors that were rejected by senior bureaucrats as 
having any substantial impact on policy development practice were anecdotal stories, the 
representations of individuals, and externally produced research. Research generally, and particularly, 
external research studies were considered to be relatively minor considerations as evidence for policy 
decisions or recommendations, although it is highly probable that at least some reference to research 
would be incorporated in some forms of staff advice. 

Participants were then asked to list the factors or evidence that should be considered in 
developing education policy. The results generated a list of 151 suggested factors covering a wide 
range of potential forms of evidence; these were assigned to one of 20 response categories we 
developed during the course of data analysis. Senior bureaucrats placed highest value on 
corporate/pragmatic evidence - including financial considerations, efficiency and policy 
implementation factors, and the direction of government’s strategic plan — together comprising 
more than 42 percent of the factors cited. In order of descending frequency, the following categories 
of evidence were listed as important to policy decision-making: (a) financial considerations/ 
efficiency/capacity to implement (21%); (b) government/department’s strategic goals/direction 
(21%); (c) education research or indicators (15%); (d) impact on students/clients (13%); (e) potential 
for social and economic development of the province (9%), and; (f) representations/pressure from 
special interest/stakeholder groups (5%). Among the categories of evidence least cited were (a) 
professional and personal knowledge, values and beliefs; (b) government/minister’s reputation; (c) 
equity and fairness considerations; (e) linguistic and cultural considerations; (f) avoidance of negative 
media attention; (g) jurisdictional considerations; and, (h) the minister’s direction, together 
comprising about eight percent (8%) of factors/evidence cited. 


3 Or since they had been in their current position, if it had been less than 12 months 

4 Corporate/ pragmatic influences are defined as practical, ideological, strategic and financial considerations. 
This cluster includes: department’s strategic plan; budget considerations; cabinet/government’s overall 
direction, and; potential to directly contribute to economic development of province. 

5 Political-democratic considerations are defined as those factors that relate to the status, image or reputation 
of the government or minister. This cluster includes: potential to advance the reputation of the 
minister/government; minister’s particular interest of direction; representations of special interest and 
advocacy groups; public opinion, and; the representations of the mass media. 
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Table 3 

Senior bureaucrats’ Perceptions of the Relative Influence of Different ¥ actors/ Evidence on their Recent Policy-relevant 


Decisions / Recommendations (N—148) 


Factor/Evidence 

Mean 

Standard 

Deviation 

Advice of staff and/or colleagues 

1.80 

1.29 

Past experience with this issue or with a related issue 

1.34 

1.59 

Personal or professional beliefs and values 

1.16 

1.53 

Department’s strategic plan 

1.15 

1.56 

Budget considerations 

0.86 

1.92 

Research or indicators prepared in-house by department staff 

0.86 

1.70 

Cabinet/Government’s overall direction 

0.73 

1.69 

Potential to advance the reputation of the Minister/government 

0.72 

1.64 

Minister’s particular interest or direction 

0.59 

1.79 

Representations of other (non-business) groups 

0.22 

1.95 

Public opinion 

-0.13 

1.75 

Potential to directly contribute to economic development of 

-0.23 

2.16 

Province 



Provincial government commissioned research or assessments 

-0.37 

2.20 

Avoidance of negative media attention 

-0.54 

1.96 

Representations of individuals (students, parents, educators) 

-0.57 

2.07 

All Research 6 

-0.82 

2.04 

Representations of business groups or the private sector 

-1.19 

1.82 

External Research Studies 7 

-1.36 

1.88 

Situation or event someone told you about outside your 

-1 65 

1 73 

professional environment 

JL ■ U A 

-L • / A 


Finally, participants were asked to provide some examples of research or indicators that they 
remembered or had used in their policy development practice. The findings presented in Table 4 
suggest that senior public servants value survey research and large-scale assessments over qualitative 
research. The vast majority of examples of research and indicators that they remember or have used 
in policy development are large-scale quantitative studies such as provincial, national and 
international assessments, educational finance studies, demographic research and labour market 
outcome studies. 


6 All Research includes: University Research (Education); University Research (other than Education); 
Research studies by other agencies; CMEC Research and Assessments; Federal Government Research; 
Research or indicators prepared in-house by department staff; and, Provincial Government- commissioned 
research or assessments 

7 The following categories were collapsed to form the External Research category: Research studies by other 
agencies (Conference Board, Fraser Institute, etc.); University Research (Education); University Research 
(other than Education); CMEC Research and Assessments; and, Federal Government Research 
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Table 4 


Categories of Research I Indicators Senior bureaucrats Recall or Have Used in Policy Development (N—51) 


Description 

Frequency 

Percent 

Provincial/national/international achievement and literacy 

15 

29.1 

assessments 

Educational finance studies 

9 

11.4 

Demographic studies (e.g., enrolments, capacity) 

8 

10.1 

Government-commis sioned res earch 

8 

10.1 

Post-secondary labour market outcomes 

7 

8.9 

Participation/graduation indicators 

6 

7.6 

Social/financial Indicators (e.g., health/ income support/ 
employment) 

5 

6.3 

Aboriginal achievement and attainment indicators 

4 

5.1 

Minority language indicators 

3 

3.8 

OECD research/indicators 

2 

2.5 

Other external research 

2 

2.5 

Busing studies 

1 

1.3 

School or school district-based research 

1 

1.3 

Total 

79 

100 


Profile of participants: Ministers of Education. The participant group of ministers of 
education comprised two females and six males, all over 45 years of age and all veteran 
politicians, each having served more than two terms in office. In general, however, they had a 
relatively short tenure in their education portfolios — four or fewer years. 

Evidence and policy practice: Interview data. Ministers of education were asked a 
series of questions relating to their policy development practice and the role of research in 
decision-making. The findings suggest that political decision makers vary considerably in their 
perspectives, but generally have very limited direct knowledge of education research. Some 
ministers responded to questions that probed their understanding of research-based evidence by 
referring to the importance of experimental studies, using terms like ‘double-blinded’ and ‘solid 
data’. Others associated valid research with program evaluations. Most participants expressed 
the view that valid research should be objective and involve “many cases” so it can be 
generalizable to the population. Ministers’ responses to questions that probed their 
understanding of the relationship of current educational research to policy decisions were 
divergent and confused, suggesting that the political decision makers in this study have a very 
superficial awareness of current research and its potential use for educational decision-making. 
Ministers were clearer in their representations of the value of external or university-based research 
and expressed negative views with respect to accessibility. The consensus view of the participants 
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interviewed was that university research is obscure and inaccessible to policymakers. The following 
exchange illustrates this point: 

M2: Once the research is being done it’s just sitting there; it’s staying in academia. 

It’s not moving out into government, into communities; into business, as it should 
be. Now I’ve seen some instances where it’s been successfully done, but I have to 
admit, that from the year and a half that I’ve been here I’m struggling very hard to 
find an instance where it’s been useful to me in my work. 

R: So, in order for it to be more useful you’d say it would have to be more relevant 
to the issues that are being dealt with by government? 

M2: You see it could be that the research they are doing is relevant, but we don’t 
know about it. They’re just not letting us see the results of their research; they’re not 
saying to us, “This is what we have done. Could it be of use to you?” We’re not 
involved, I guess. That’s what [the problem] would be. 

The knowledge generated by university research was also seen to be inconsistent with the 
information needs of government. Participants questioned the relevance of research to the day-to- 
day policy questions that are important to education ministers. As one minister noted, education 
research is ‘not a main reason for change’. The following excerpts are typical of these 
representations: 

M7: Well, I haven’t had a chance to look at anything [university researchers] are 
doing so I really don’t know, but I do find that sometimes, just from past 
experiences, that the research coming out of (names university) is not always; well 
not the research, but some of the conclusions they draw from their research is not 
always applicable.... 

M2: Well, I don’t really know about it, to be truthful. As I was saying earlier that’s 
one of the things that is most surprising to me; that with the amount of research that 
is done at the university, how little discussion or impact it’s had on my work as the 
Minister in this portfolio. 

Participants also perceived a wide separation between the education research community and 
government departments where education policies are conceived. Participants observed that 
education researchers operate in different operational environment with a vastly different set of 
goals and orientations. Some ministers charged researchers with criticizing the education policy 
directions of government instead of offering productive suggestions or engaging in constructive 
debate. Several ministers expressed the view that university researchers tend to either ignore 
important policy problems or simplify those problems by remaining focused at a theoretical or 
ideological level, while failing to account for real-life policy pressures: 

Ml: I guess that what I find with university research is that those who are doing it 
appear to be somewhat removed from the immediacy of it all; it’s not engaged, as it 
were, and [university researchers are] somewhat reluctant to engage those who are 
involved in a particular issue on a daily basis. That’s being pretty general; that’s not 
the case with all university research. 

M3: University research? Well, I don't know... we had all the mathematics people 
look at the mathematics curriculum in (province). They did that in 1998, 1999, 2000, 
as a result of our kids doing poorly in mathematics and so we had all these pundits. 

They're all experts. They put together this curriculum. The curriculum came out 
and.. .some of the very professors who were involved in the generation of that 
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[curriculum] came out and criticized it in the papers. So I don't understand where 
they're coming from. 

In each of the interviews, ministers were then asked to identify from a list of possible influences or 
evidence, the five factors they consider to be most influential when making education policy 
decisions. Collectively, participants identified a total of 42 factors 8 represented in Table 5. 

These data support the findings from the in-depth interviews. While ministers claim to value 
a broad range of policy evidence, there was a collective preference for policy evidence that falls 
within five principal categories of evidence: (1) political-democratic factors, which we frame as 
inclusive of public opinion, the representations of individuals, special interests/advocacy groups and 
the representations of the media; (2) pragmatic or practical considerations, such as financial 
concerns and governance ideology; (3) global (macro-level) competitive influences and pressures - 
in particular, evidence such as educational rankings or standings based on large-scale national and/or 
international assessments; (4) staff advice; and (5) personal and/or professional beliefs, values and 
experiences. 


Table 5 

Education Ministers’ Perceptions of Most Important Influences or Evidence on which to base Policy Decisions (N=8) 


Factor/Evidence 

Number of 
References 

Government’ strategic direction 1 ’ 

8 

Representations of individuals or special interests 1 " 

7 

Advice of the deputy minister and senior staff 

6 

Department’s own research and indicators 

5 

Public opinion 

5 

Budget considerations 

4 

Personal or professional beliefs, values and experience 11 

3 

Province’s goals for economic growth 

1 

Potential to advance the reputation of the government or the minister. 

1 

Minister’s own particular interest or emphasis 

1 

CMEC research and assessment(s) 

1 


Ministers compared with senior education bureaucrats. There are striking 
similarities in how politicians and bureaucrats represent the importance of different categories 
of policy evidence, but there are differences in the relative standing or ‘ranking’ of each 
category. In Table 6 the key policy drivers for politicians and senior public servants are grouped 
into five main categories, ranked, in order of influence. As noted, for politicians the most 
significant influence on policy development seems to be based largely on broadly defined 


8 In two cases participants identified six (6) factors 

9 The following categories were collapsed: Cabinet/Government’s overall direction and the Department’s 
strategic plan 

10 The following categories were collapsed: representations of individual students, parents or constituents; 
representations of business groups and the private sector; representations of individual 
teachers/instructors/administrator, and; Representations of special interest stakeholder groups. 

11 The following categories were collapsed: past experiences with related issues and personal or professional 
beliefs and values 
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political-democratic influences: public opinion, advocacy, the mass media and other political 
and pragmatic pressures, including political ideology. While senior bureaucrats also consider 
political-democratic considerations to be important, these factors are ranked lower. Deputies 
and assistant deputies place highest value on the internal advice provided by staff and 
colleagues, followed by past experience, personal and professional beliefs and values, corporate 
and pragmatic evidence and, political considerations. While politicians deem staff advice to be 
important, it was considered to be less important than corporate and pragmatic considerations 
(such as financial exigencies and governance ideology), and global (macro-level) competitive 
pressures. 

Table 6 


Comparison and Ranking of Most Important Categories of Evidence for Education Polity Decisions, Education 
Ministers and Senior Education Bureaucrats 



Education Ministers 


Senior Education Bureaucrats 

1 . 

Political Considerations 

1 . 

Advice of Staff and/or Colleagues 

2. 

Corporate and Pragmatic Considerations 

2. 

Past Experience 

3. 

Global (Macro-level) Competitive Pressures 

3. 

Personal /Professional Beliefs and Values 

4. 

Advice of Staff 

4. 

Corporate and Pragmatic Considerations 

5. 

Personal /Professional Beliefs and Values 

5. 

Political Considerations 


The 2012 Study 

Profile of participants: School trustees and superintendents. Most survey 
respondents (62%) were from Western Canada, with 22 percent from the Central region 
(Ontario and Quebec), and 13% from the Atlantic Provinces. Three percent of respondents did 
not identify their location. The balance of male and female respondents was similar with slightly 
more females (53%) than males. The age of respondents varied from under 30 years (1.5%) to 
over 60 (33%) with the majority (67%) over the age of 50. About two thirds of respondents 
reported that they had served in the role of trustee for more than five years. School trustee 
occupations were quite varied (e.g. stay-at-home parents, skilled tradespersons, farmers, fishers, 
educators, office workers, business owners, health care professionals, clerks, civil servants, 
managers/executives and retirees). 

Evidence and policy practice: Survey data. We first sought to understand the general 
categories of school board decisions made by trustees in the fulfillment of their governance 
roles. Their responses, summarized in Table 7 provide a glimpse of the decision-making 
priorities of school boards across the country. It appears that the primary focus of school board 
policy decision-making relates to programs, school development, and assessment. The second 
most commonly identified category was actual engagement in policy development and 
governance. Other common foci included the provision of safe caring schools, transportation, 
school restructuring, human resources, technological innovation, strategic planning, engaging 
families and community, budgeting, and resource allocation. 

Participants were also asked to indicate their views on a series of statements relating to 
governance and decision-making by using a seven-point Likert scale (strongly disagree to strongly 
agree). Most trustees/superintendents (56%) either agreed or strongly agreed that “major decisions 
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in this school district are based on data,” while 33 percent were less definitive in their responses 
indicating that they“ somewhat agree” with the statement. These responses suggest that although 
Table 7 


Categories and Frequency of Policy Decisions Recalled by School Trustees and Superintendents 


Category 

Frequency 

Percent 

Programs, school development, assessment 

78 

25.6 

Policy development and governance 

48 

15.7 

Safe, caring environments 

28 

9.2 

Transportation 

26 

8.5 

Restructuring and school closures 

18 

5.9 

Internal board/district matters 

17 

5.6 

Human resources 

16 

5.2 

Technology and Innovation 

16 

5.2 

Strategic planning and oversight 

14 

4.6 

Family and community 

13 

4.3 

Budgeting and finance 

12 

3.9 

Maintenance and infrastructure 

5 

1.6 

Ministry of Education 

5 

1.6 

Aboriginal/First Nations 

4 

1.3 

Religion 

3 

1.0 

Public relations 

2 

0.7 

Total 

305 

100 


reliance on data may not be an established approach to all decision-making in school boards, a 
majority of participants feel that it has become a meaningful aspect of their decision-making 
practice. To obtain a more specific understanding of the factors that influence school board 
decision-making, we presented tmstees with 20 potential factors/influences and asked them to 
indicate on a seven- point scale, the extent to which each has had an effect on specific decisions or 
recommendations that they have made as trustees (Table 8). The principal six factors are as follows: 
(1) personal or professional beliefs and values, (2) potential to directly influence student 
outcomes/student learning, (3) advice of district staff and/or colleagues, (4) the school board's 
strategic plan and (5) past experience. Trustees and superintendents rated the following six factors as 
having the least influence on decision-making: (1) representations of business/private sector, (2) 
pressure from special interest or lobby groups, (3) a situation or event someone told you about, (4) 
pressure from government (ministry/department of education), (5) public opinion/avoidance of 
negative media attention, and (6) university-based research. Analysis of the data by province showed 
very little variance in the factors and evidence that influence school board decision-making across 
jurisdictions. 

Other factors and evidence with a moderate level of influence include (1) the potential to 
advance the reputation of the school district or board, (2) research or indicators prepared in-house 
by district staff, and (3) budget considerations while factors such as the representations of teacher 
and administrators, business/private sector interests, ministry of education assessments and other 
external research were seen as having only marginal influence on school board decisions. 
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Table 8: 


School Board Trustee and Superintendent Perceptions of the Relative Influence of Different Factors/ Evidence on their 
Recent Policy-relevant Decisions/ Recommendations 12 


Factor/evidence 

N 

Mean 

Standard 

Deviation 

Personal or professional beliefs and values 

337 

1.70 

1.39 

Potential to directly influence student outcomes/student learning 

333 

1.66 

1.81 

Advice of district staff and/or colleagues 

340 

1.28 

1.56 

School board strategic plan 

337 

1.17 

1.84 

Past experience with this issue or with a related issue 

334 

0.96 

1.74 

Potential to advance the reputation of the school district or board 

338 

0.86 

1.92 

Research or indicators prepared in-house by district staff 

334 

0.74 

1.89 

Budget considerations 

337 

0.55 

2.06 

Representations of teachers and/or administrators 

334 

0.23 

1.95 

Representations of business/private sector 

337 

0.15 

1.95 

Ministry/department of education strategic plan/overall directions 

335 

-0.06 

1.98 

Research conducted by outside agencies 

333 

-0.29 

2.18 

Provincial government research or assessment information 

335 

-0.35 

1.96 

Decisions of other school boards 

337 

-0.69 

1.88 

University-based research 

333 

-1.08 

1.97 

Public opinion/avoidance of negative media attention 

335 

-1.09 

1.83 

Pressure from government (ministry/department of education) 

336 

-1.15 

1.96 

Situation or event someone told you about outside your professional 

323 

-1.24 

1.90 

environment 




Pressure from special interest or lobby groups 

338 

-1.26 

1.77 

Potential to advance reputation of district/board 

330 

-1.53 

1.68 


Evidence and policy practice: Focus groups. The focus-group protocol was designed 
as an in-depth probe of the factors influencing policy decision-making. Accordingly, we were 
able to assess this measure separately for school board trustees and superintendents. In each 
focus group session we presented each participant with a list of potential factors and influences 
on decision-making (similar, but not identical to the list of factors provided on the 
questionnaire). The list was a slightly modified form of the instrument used in Galway’s (2006) 
study. We asked trustees and superintendents to identify and rank, in order of importance, the 
top five factors/influences on their own policy practice. Although the two lists were not 
identical, findings from the focus group data with trustees and superintendents generally 
paralleled the findings based on the survey data with some minor variations. 

Trustees tended to place more value on the “representations of students, parents and 
constituents” than did superintendents while superintendents ranked “past experience” and 
“external research” somewhat higher than did tmstees. Otherwise there was close alignment 
between the rankings of trustees and superintendents with the following factors all ranked high: (1) 
advice of the CEO (superintendent) and senior staff, (2) school board’s own research and indicators. 


12 Mean values for each of the statements were converted from a 1 to 7 interval scale to a -3 to +3 interval 
scale, where -3 represented 'did not at all influence', 0 was neutral and 3 represented 'influenced to a great 
extent'. 
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(3) budget considerations, and (4) government/department of education’s overall direction. The data 
derived from the focus group responses were somewhat similar to the questionnaire data, with some 
notable exceptions. For example, the questionnaire data showed the influence of the ministries of 
education to be a relatively minor influence while the focus group participants considered it to be 
much more significant. The lowest ranked factors and evidence for both groups were (1) 
representations of individual teachers and administrators, (2) representations of special interest 
groups and (3) factors such as “anecdotal reports” and “economic goals for the region.” 

With regard to research utilization, our focus group sessions with school board members in 
some regions revealed a somewhat different perspective - not entirely consistent with the pan- 
Canadian survey data. In Quebec, commissionaires (trustees) noted that frequently they work closely 
with university researchers in determining policies and programs. The following response to our 
question, “What kind of research, if any, do you value when making policy decisions?” illustrates: 

It depends on the policy. Our board has a policy on policy development. We don’t 
research that very much. But when we have our policy for dealing with children with 
special needs then we use a lot of research of our university partners, McGill and 
Concordia. Doing our policy on digital citizenship.. .we took a lot of research from 
the Internet [and] we went out to people who implemented things before. Our policy 
on fieldtrips .. .is based on the law and insurance requirements, so it really depends 
on the nature of the policy, but we will do a fair amount of research, as appropriate. 

Another focus group of school board trustees in Saskatchewan echoed these views; their perspective 
suggesting that they place higher value on research evidence for decision-making than would be 
indicated by the data derived from the questionnaire: 

I think that’s what validates. I mean that’s where.. .you can separate the politics and 

[decision-making] becomes evidence-based_When you do that then you’ve got 

substance based on evidence and people can stand behind it. It’s not so much public 
opinion and private opinion; it’s more a matter of some form of evidence-base. 

In one of the Atlantic Provinces, one participant responded, “ [i] t depends on the issue. If it is 
something to do with student achievement then I think we look at university research, scholarly 
work and evidence more closely than we ever have.” 

These responses suggest that school board tmstees, like education ministers and senior 
education bureaucrats place high value on political considerations in decision-making, such as the 
representations of students, parents and constituents. Their responses also correspond to the 
representations of ministry of education policy-makers in that they place high priority on staff advice 
and past experience. The responses of the participants from the two studies are also similar in that 
they say they place substantial importance on personal and professional beliefs and values, corporate 
and pragmatic influences, such as budget considerations and the government’s “overall direction.” 

The value placed on external research as policy evidence was mostly consistent among the 
participants in the two studies. Generally, the questionnaire data indicates a low level of research 
utilization among Canadian education policy elites. The consensus view of the political decision¬ 
makers interviewed in the 2006 study is that external research, and particularly university research is 
somewhat obscure and inaccessible to policymakers - a scenario where education researchers make 
a very marginal contribution to education policy decisions. The representations of school board 
trustees and superintendents were generally consistent with this perspective, although less definitive. 



Education Policy Analysis Archives Vol. 23 No. 109 


24 


The mean scale score 1 ’ representing the influence of university research on policy decision-making 
within ministries and school boards was -1.36 and -1.08, respectively. Based on focus group data, 
there was some variation in these findings for certain regions of the Country where school board 
trustees and superintendents said they relied on universities to undertake policy research and were 
likely to utilize research evidence if they were aware of its existence and deemed it relevant to their 
circumstances. It also appears that for policy decision-making, not all sources of research evidence 
are given equal sway. In both studies, internal research and indicators prepared either by ministry 
staff or school district staff was ranked the same - in the middle of the positive range of responses 
(mean scale scores of 0.86 and 0.86, respectively). The data show that external (university-based) 
research, however, did not rank highly in either study. 

Discussion 


Theorizing Low Research Impact 

Several educational researchers have described the educational landscape in Canada over the 
past quarter century as turbulent and uncertain as policymakers have tried to negotiate the problems 
of fewer students, unstable budgets, and new expectations for schools (Levin, 2003b; Levin & Riffel, 
1997; Sheppard, Galway, Brown & Wiens, 2013; Ungerleider, 2003). There is widespread agreement 
that decision-making in education now occurs under a microscope; it has become highly politicized 
and subject to intense media attention (Levin, 2004a; Lingard & Rawolle, 2004; Thomson, 2004; 
Ungerleider, 2004). As Beck (1997) and Giddens (1990; 1994; 2003) argue, cultural changes and the 
growing complexity of society creates greater levels of uncertainty, resulting in people becoming 
more restless, taking less for granted, and initiating action to reduce their perceptions of risk. 
Canadians are passionate about education, and are unabashed about raising the bar for what they 
expect from schools. But public expectations for education are constantly shifting, creating policy 
challenges and dilemmas for decision makers. More than in the past, citizens and education lobby 
groups are better positioned with knowledge and expertise; they are better equipped to challenge 
institutional experts within school boards and ministries of education and have a greater capacity to 
lobby governments and force change. 

As Guppy and Davies (1999, p. 278) point out, parents and educational interest groups are 
“less deferential towards the educational establishment and are more likely to feel entitled to 
challenge experts.” Based on a review of two decades of polling evidence. Guppy and Davies (1999) 
showed that public confidence in Canadian schooling had declined in all population subgroups. A 
more recent survey, undertaken by the Canadian Education Association, showed that between 1984 
and 2007 the percentage of Canadians who reported a great deal or fair amount of confidence in 
schools dropped by 31 percentage points from 76 to 45 percent (Canadian Education Association, 
2007). This dissatisfaction has been expressed in myriad ways - from a startling increase in private 
tutoring (Davis, Aurini & Quirke, 2002) - particularly among wealthy Canadians (Cain, 2013) - to a 
rise in the number of legal cases against school boards (Sheppard, 2012). A 2012 Ipsos-Reid poll 
revealed that Canadians gave a poor rating to the “overall state” of the public elementary school 
system. More than half of those polled (58%) held the view that a private school education is better 
than a public school education and about two thirds (63%) said they would choose a private school 
education for their children, if money were no concern. 

At the same time, virtually all regions of the country have been experiencing major 
demographic changes leading to shifts in student populations, profound enrolment decline and 


13 where -3 represented 'did not at all influence', 0 was neutral and 3 represented 'influenced to a great extent'. 



Research and evidence in education decision-making 


25 


greater diversity in the school-aged population. Census data show that Canada recorded the highest 
population growth among G8 nations in 2013-14; however, net international migration accounted 
for almost two-thirds (66.5%) of the total increase. In the last several decades there have also been 
profound changes in proportion of Canadians leaving smaller towns and settlements for urban areas 
- from 62% in 1951 to 81% in 2011 (Statistics Canada, 2011). Within the country the population 
share of some regions, notably Quebec and the Atlantic Provinces has decreased over the past three 
decades while Ontario and the Western Provinces has increased (Statistics Canada, 2014). For, 
example, over the twenty-year period, 1993-94 to 2013-14, the number of school-aged children in 
the Atlantic region fell by 30 percent or about 1.6 percent per year and enrolment losses between 
one and two percent per year are expected for at least the next decade (Galway, 2015). 

Schools are among the first public institutions to be impacted by socio-cultural and 
demographic changes. Addressing sustained enrolment changes (influx or decline) and responding 
to more diverse student populations require considered policy responses. Other social tensions and 
technological changes are similarly first felt in schools and colleges, leaving policy actors to negotiate 
everything from school closures and realignment to safe school protocols and diversity policies, 
often with a restless and impatient public. Ministries of education and school boards are expected to 
base their policy decisions on the best evidence available to them. The findings from these studies, 
however, portray the policy development process as ambiguous and political; driven by informal 
evidence that is only loosely connected to the educational research community. 

These findings are consistent with the theoretical orientations of Beck (1994; 1997), Majone 
(1989), Kingdon (1995), Levin (2003b; 2004b) and Stone (2002), all of whom conceptualize public 
policy-making as divergent, uncertain and risky - driven by personal, political, and experiential 
influences. In a politicized environment, independent university research, especially that which 
might be tacitly critical of existing policy, may either represent additional political risk or be 
perceived as inaccessible or irrelevant to settling urgent policy questions. Under such conditions, the 
values, reward systems and accountabilities against which social scientists operate are likely to differ 
substantively from those of system-level decision makers, thereby creating a policy environment that 
might be antagonistic to external research. These conditions may well force policymakers to set aside 
research-informed policy (which is normally a longer-term proposition) in order to attend to less 
risky and more immediate pragmatic and democratic/political considerations. 

Making Research Matter More 

These studies raise questions about whether the existing decision-making approaches are 
useful in advancing a sustainable policy agenda for Canadian schools. They also underscore the 
importance of undertaking policy work in consideration of the best available research. Decision 
makers need ready access to research evidence to guide their practice in the public sphere, but 
perhaps more pointedly, to refute incorrect information and hearsay that can so often form the 
substance of unqualified demands for educational change. Following Edwards et al. (2007), Jbilou et 
al. (2007) Landry et al. (2001), Nutley et al. (2008) and others, we propose that to attenuate the risk 
associated with greater reliance on education research stronger connections must be fostered 
between the research community and several levels of the bureaucracy within school districts and 
ministries of education. In short, to become a more significant consideration in the policy advice 
senior advisors provide to politicians, research must become more familiar and trustworthy. 

The importance of tmst and confidence has been discussed elsewhere in the literature (e.g., 
Nutley et al., 2009; Tseng, 2012). One of the key questions raised by researchers is how do 
policymakers and other information consumers sort through competing claims in guiding their 
practice (Lubinenski, Scott & Debray, 2014). While the studies compared here have focused on the 
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standing of research and other factors as policy evidence, several researchers have begun to 
conceptualize research utilization as a social process: “one that unfolds within a complex ecology of 
relationships, organization and political and policy contexts” (Yohalem & Tseng, 2015). One recent 
US study of school board decision makers found that the value of research could not be decoupled 
from trustworthiness of its sources and the motives of the person conveying the evidence (Asen, 
Gurke, Donners, Solomon, & Gumm, 2012). Rosenblatt and Tseng (2010, p. 201) have argued for 
the need to understand the importance of the demand side of research-policy equation, including 
“user settings and perspectives, political, economic and social contexts and the various uses of 
research.” 

Based on large-sample studies in the Canadian health care sector several strategies have been 
suggested to improve reciprocal understandings between the research and policy communities. 
Landry et al. (2001) identified several determinants of research utilization including the mechanisms 
linking researchers to users and the user’s operational context. Jbilou et al. (2007) added that 
communication networks, partnerships and other links between researchers and decision-makers 
could help effect a cultural change in governmental organizations - whereby there is greater support 
and value for research into practice. Programs such as the former TLRP in the UK and the former 
Community-University Research Alliance (CURA) program in Canada represented efforts to 
support alliances between community organizations and universities to foster innovative research, 
training and the creation of new knowledge (James & Pollard, 2011; Social Sciences and Humanities 
Research Council, 2013). More to the issue of engendering trust and confidence, Edwards et al. 
(2007) describes such programs in terms of a reciprocity or co-construction of research-based 
knowledge that serves to connect organizational boundaries and strengthen the warrant for research 
studies. In the following we draw upon the findings from these studies and the work of others 
researching this area to suggest some modest proposals for how such evidence could become a 
more integral aspect of education policy and practice. 

Engaging ministry and school board advisors. Following on Jbilou et al. (2007), we 
see research collaborations, research forums and regular formal (or informal) meetings with 
mid- and senior-level school board and ministry officials as important mechanisms for 
interchanges between knowledge producers and the education policy community. There are few 
opportunities for external research to reach policymakers, except through the bureaucracy and, 
to lesser degree, through the efforts of special interest groups and the mass media. Reimer and 
McGinn (1997) wrote about the difficulty they experienced in communicating research findings 
to government decision makers. As a dissemination strategy, direct access to school board 
trustees and/or ministers of education is problematic for several reasons. Access to policy elites 
is difficult at the best of times and focusing the attention of elected officials on complex 
teaching and learning issues is a longer-term proposition. Even in instances where researchers 
are successful in engaging elected decision makers, due to their crowded schedules, such 
meetings are likely to be short, infrequent and limited in their long-term impact. A more 
effective strategy would involve the active engagement of mid to senior policy professionals 
who prepare the information — informal e-mails or briefings, decision notes and policy papers — 
that is circulated ‘up the line.’ 

Bah Diallo (2001), a former minister of education in Guinea, says that politicians can only 
effectively initiate policy change with the cooperation of the bureaucracy. Administrators identify for 
ministers not only what is needed to effect change, but also the appropriate pathways to 
implementation: 

Administration constitutes the passage oblige , the gateway, the bridging link between policy 

and practice, between political intent and the hard reality of day-to-day business. As such it 



Research and evidence in education decision-making 


27 


plays the difficult role of reconciling the need to maintain the system and the need to lead 
the changes and the reforms. [...] administrators have a tremendous influence on the system 
processes and the policy-makers’ capacity to decide because administrators can influence 
action where it counts the most, that is, in the field, where tilings happen, (p. 22) 

The policy actors who participated in the 2006 and the 2012 study indicated they place high value on 
knowledge produced or validated by insiders - ministry and/or school district policy staff, a finding 
consistent with Majone’s (1989) suggestion that the policy analyst plays a pivotal role in determining 
what evidence is considered by policy elites and the manner in which policy advice is constructed. 
While mid-level bureaucrats and district program specialists represent the entry point for external 
knowledge, senior bureaucrats mediate the cascade of information to elected decision makers by 
distilling and validating evidence and bringing together research with other forms of evidence to 
generate ‘trustworthy’ staff advice. 

Intersecting communities of practice. There are several small-scale efforts that could 
be undertaken to create intersections between the individual communities of practice of 
researchers and policy advisors/practitioners. Formal and informal professional exchanges could 
create reciprocal space for discussion of policy issues and problems and the lands of research 
that could contribute to public discourse and/or policy deliberations. In our earlier work on 
researcher perceptions of evidence used in policy making (Galway, 2006), participants 
referenced the challenges in sustaining regular professional exchanges between academics, senior 
public servants and ministry advisors, but also their effectiveness in connecting the three 
communities of practice. 

Another point of interchange could be the establishment of rotating research chairs within 
education ministries and faculties of education - whereby mid-to-senior level public servants could 
become immersed in the research culture of the university while academics could experience the 
policy environment of government. There is already some precedent for this kind of exchange. In at 
least one Canadian province there is a standing research fellowship that enables senior executives of 
the government to work in a university setting and to teach and research in their professional 
discipline. Through the fellowship, there have been similar moves in the opposite direction, whereby 
faculty members take up residence in government departments. For example, the former Canadian 
Council on Learning had conceived a plan centered on the establishment of ‘researchers in 
residence’ as a means of infusing research into policy discussions at the level of education 
ministries 14 . Under such an arrangement the ministry and/or the university would underwrite or 
partially underwrite the associated costs of the exchange; the participants would then focus on one 
or more policy questions occupying the district/ministry agenda. This research would imply that 
these and similar opportunities for researcher-bureaucrat exchange should be encouraged as a means 
of creating common understandings among school districts, government departments and education 
faculties. 

Another approach formalizes the involvement of educational leaders in research through the 
promotion of advanced professional degrees aimed at administrators, bureaucrats and senior 
educators. Now reasonably well established in many faculties, the professional doctorate requires 
that candidates undertake research relating to their professional practice. While relatively few 
professional doctorates are awarded each year in relation to the number of professional staff 
working in education, the connection of policy analysts and program specialists to the research 
community through professional learning opportunities would appear to be an effective, albeit 


14 Personal communication 
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small-scale means of infusing research into policy practice. School boards, post-secondary 
institutions and ministries of education could provide incentives to encourage professional staff to 
undertake research-based professional degrees as a means of professional development. Financial 
incentives such as grants to assist with tuition and research expenses, sabbatical leaves for senior 
public servants and/or promotion incentives could entice more senior education officials to begin 
doctoral or professional masters programs. Moreover, the design of many professional doctorate 
programs affords the student considerable flexibility. The completion of degrees, partially through 
Web-based study, the ability to study part-time over extended periods, and the offering of 
compressed didactic components using itinerant professorial instmction all contribute to the 
professional degree option as a viable means of making research more familiar to the policy 
community. 

Moderating the critical stance of the academy. We have argued that research will 
become more entrenched in policy development if it can become more familiar and less risky for 
policy makers. The data from the 2006 study show that ministers of education represented 
researchers as sometimes adversarial - either indifferent to the pressing educational problems on 
the government’s policy agenda, or too often, critical of education policies and practices. 
Recognizing the cautions of Pirrie, Adamson and Humes (2010) regarding the spread of 
bureaucratic rationality and the responsibilities of academics to speak truth to power and 
preserve freedom in academic life, we believe some consideration should be given to the ways in 
which academics represent their research in the public sphere, and how they frame policy 
critiques. While certain traditions of inquiry lend themselves to critical commentary on social 
institutions and processes (Creswell, 1998), the findings from this study suggest that sustained 
critique of policy decisions may serve to position academic research as foreign and threatening 
to policymakers. When academic researchers publicly argue for a strict (research) evidence- 
informed approach to policy, they may inadvertently overlook the political dynamic of a policy 
situation, and situate themselves as adversaries. As Caplan (1979, p. 461) writes, [t]he need for 
reciprocal relationships between knowledge producers and knowledge users in policy-making is 
clear, but the problem of achieving effective interaction of this sort necessarily involves value 
and ideological dimensions as well as technical ones.” The studies discussed here show that 
decision-makers make choices in the context of an overwhelming and widely diverse set of 
public expectations. Politicians, in some instances, seem to identify those in the academy with 
risk and uncertainty. For some researchers — particularly younger scholars — the contextual 
background to the policy process may represent uncharted territory. In order for education 
research to begin to make a difference in the policy decisions of policymakers, academics may 
wish to evaluate how far critical approaches have advanced the influence of research on the 
public policy agenda and consider different ways and means of drawing attention to their policy 
concerns. Researchers may better situate themselves to influence policy decisions by accounting 
for the ambiguities and political pressures of decision-making in their critique, while advocating 
for reasoned, research-informed policy. 

Balancing academic interests with research impact. The findings from these studies 
also suggest that attempts to stimulate more policy-relevant research will also come with the 
need to re-evaluate traditional ideas about the academic utility of research in relation to its 
practical value to consumers and policy makers. From the perspective of academic progression, 
researchers have good reason to doubt the value of undertaking commissioned or 
government/district-sponsored research. Levin (2004b, p. 9) has remarked that, “university- 
based research is driven by the university culture and reward system; it is primarily aimed at 
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communication with other scholars and the rewards are related to research grants and peer 
recognition.” In her case study of a large Canadian community-university research collaboration, 
Tapp-Neville (2015) reported that researchers were concerned that certain forms of 
collaborative and commissioned research have a lower standing than independent research as 
evidence of contribution to scholarly inquiry. If university communities, even passively assign 
lower status to commissioned policy work, they may, unintentionally, restrict the likelihood that 
collaborative research programs will attract young, able scholars. Moreover, if commissioned 
education research has limited standing within the academic community it might only be 
conducted in the ‘sidelines’ — something done to sustain a stream of research funding or 
otherwise left to researchers who have advanced in their careers and have already established 
themselves as credible academics. 

From the perspective of enhancing research impact, there is reason to be concerned about a 
culture that draws a distinction between the importance of research that is needed and valued by 
ministries of education and school districts and other ‘legitimate’ research that is valued within the 
academy. For education research to become a more visible presence in day-to-day policy discussions, 
and for the quality and amount of commissioned research to improve, consumer-driven research 
needs be recognized for the purposes of academic reputation and tenure-track promotion. Finally, if 
commissioned research is assigned a lower status with respect to promotion and tenure, it may 
unintentionally restrict the likelihood that such research opportunities will attract young, able 
scholars. This topic will not be further addressed here, although it is important area for future 
research. 


Conclusions 

On balance, the findings from the 2012 study of school board trustees and superintendents 
are generally consistent with those of the 2006 study of education ministers and senior education 
bureaucrats. We found that elected officials and their advisors in both contexts use a variety of 
forms of evidence in their decision-making, including non-formal sources of evidence such as local 
(internal) knowledge and experience. Policy elites, particularly at the ministry level, tend to place 
considerable reliance on political-democratic forms of policy evidence, followed by staff advice, 
corporate and pragmatic considerations, personal and professional beliefs and values and past 
experience. Both participant groups also confirm the influence of formal sources of information, 
such as research evidence; however the form of such research matters, and appears to be largely 
limited to internally generated research and indicators. 

Findings from the 2006 study show that ministry policymakers express largely negative 
perspectives about the availability and value of certain forms of research-based knowledge. The 
ministers interviewed were either oblivious to the research undertaken in universities or deprecating 
in their appraisal of its availability and utility. Both political actors and senior bureaucrats say that 
they place greatest value on research-based evidence that originates or has been vetted by staff and 
policy advisors within the ministries themselves. Tmstees and superintendents expressed a similar 
perspective placing greater value on the school board’s own research and indicators, while 
downplaying university research as less influential to school board decision-making — this form of 
evidence ranking 15th of 20 possible decision-making factors. Still, school board trustees and 
superintendents were more predisposed to the notion of “data-driven” decision-making and some 
focus group participants mentioned collaboration with university researchers; however, these 
references were few and limited in scope. 
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We conceptualize use of low research impact partly in terms of trust vs. risk avoidance. In 
both policy contexts - provincial departments of education and school boards - we conclude that 
decision-making is ambiguous and risky. Policies and practices are developed and enacted in an 
uncertain policy context, often with diverse and impatient constituencies. Following Beck (1994, 
1997), in uncertain times, decision-makers value knowledge that is familiar and emerges from their 
own community. Such knowledge, while frequently originating in external studies, is contextualized 
against other policy evidence, and judgments are made regarding its relative value within the larger 
policy context. This information - now repackaged in the form of staff advice - becomes privileged 
and trusted by ministers and senior bureaucrats as authentic knowledge. The problem inherent in a 
decision-making paradigm that is overly dependent on informal, populist and experiential knowledge 
is that it is haphazard and fundamentally reactive, thereby inviting continuous reform and 
readjustment. 

The evidence and arguments presented herein support calls from others for a more open, 
collaborative, familiar and tmsted relationship between education practitioners and researchers. Such 
a change could begin to immunize schools against frequent changes in practices and policies until 
they have been referred for evaluation against the best information available. Following on other 
recent work in this area, several tangible strategies are suggested that might be helpful in effecting 
such a change. These representations also raise policy issues for administrators in some Canadian 
universities, both from the perspective of enhancing the formal and informal linkages between the 
research and policy communities, and in re-imagining the standing of user-defined (commissioned) 
research within the academy. If education research is to become a staple at the table of policy 
decision makers, those in authority within the academy would do well to examine and debate these 
issues and, where necessary, make appropriate adjustments. 
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